TRT-2796: Revert #8997 "OCPBUGS-98461: requeue CRR on transiently unavailable resources" - #9034

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@redhat-chai-bot: This pull request references TRT-2796 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "5.0.0" version, but no target version was set.

No actionable comments were generated in the recent review. 🎉

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

_{Comment @coderabbitai help to get the list of available commands.}

@redhat-chai-bot: redhat-chai-bot unauthorized: /override is restricted to Repo administrators, approvers in top level OWNERS file, and the following github teams:openshift: openshift-release-oversight openshift-staff-engineers openshift-sustaining-engineers.

Codecov Report

❌ Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 44.16%. Comparing base (3b49a4a) to head (36a3506).

@@            Coverage Diff             @@
##             main    #9034      +/-   ##
==========================================
- Coverage   44.18%   44.16%   -0.02%     
==========================================
  Files         772      772              
  Lines       96340    96340              
==========================================
- Hits        42565    42550      -15     
- Misses      50831    50841      +10     
- Partials     2944     2949       +5     

/approve

/lgtm

/verified by e2e

@jparrill: This PR has been marked as verified by e2e.

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jparrill, redhat-chai-bot

The full list of commands accepted by this bot can be found here.

The pull request process is described here

/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator

/hold
for more payload evaluation

Test Results

e2e-aws

• Status: ✅ PASS
• Started: 2026-07-18T12:14:26Z
• View Job
• View Job History

e2e-aks

• Status: ✅ PASS
• Started: 2026-07-17T13:12:31Z
• View Job
• View Job History

/retest-required

@redhat-chai-bot: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Full PR test history. Your PR dashboard.

Analysis of the two cited periodic runs

Run 1 (Jul 19 18:46 UTC): TestCreateClusterHABreakGlassCredentials fails. The CRR for sre-break-glass is stuck at:

• PreviousCertificatesRevoked=False: WaitingForAvailable(Previous signer certificate not yet revoked.)
• NewCertificatesTrusted=True: AsExpected(New signer certificate trusted.)

This means the CRR passed the propagation phase (ensureNewSignerCertificatePropagated) successfully and is stuck in the revocation phase (ensureOldSignerCertificateRevoked), specifically at the KAS trust bundle verification — the normal revocation path, not a nil-resource path.

Run 2 (Jul 19 20:28 UTC): TestCreateClusterHABreakGlassCredentials passes. The failures in this run are unrelated: Karpenter Kubelet propagation (TLS error) and NodePool Teardown timeout.

Why #8997 is not the cause

PR #8997 changed requeueSoon from false to true in 6 return paths. All 6 paths share the same characteristic: they execute only when a resource is transiently nil — the signer secret doesn't exist yet, or the trust bundle ConfigMap hasn't been created. These are guard clauses that return early before any revocation logic runs:

// ensureNewSignerCertificatePropagated
if signers == nil { return true, nil, true, nil }           // signer secret nil
if totalClientTrustBundle == nil { return true, nil, true, nil }  // trust bundle nil

// ensureOldSignerCertificateRevoked
if signerSecret == nil { return true, nil, true, nil }      // signer secret nil
if totalClientTrustBundle == nil { return true, nil, true, nil }  // trust bundle nil

The CRR in Run 1 has NewCertificatesTrusted=True, which means ensureNewSignerCertificatePropagated completed successfully — the signer secret and trust bundle both existed and were processed. The failure is in ensureOldSignerCertificateRevoked at the KAS verification step (checking that KAS has loaded the updated trust bundle and rejects the old certificate). This is the normal revocation path that was not changed by #8997.

If our nil-resource requeue was causing "excessive churn", it would only churn when the resources don't exist — which is not the case here (they exist, the propagation phase proved it). And it would fail consistently — Run 2 passing disproves systematic churn.

What the failure actually is

This is the same intermittent CRR timeout that existed before #8997 — the exact bug we were trying to fix. The CRR condition WaitingForAvailable(Previous signer certificate not yet revoked.) means the controller is polling KAS pods to verify they've loaded the new trust bundle, and this verification didn't complete within 10 minutes. This happened in 3/8 runs before our PR was merged (documented in the OCPBUGS-98461 bug report).

The correlation is temporal (our PR merged Jul 16, failure on Jul 19) but not causal — the failure path is in code we did not modify.

I'd ask to keep the hold on this revert for now while we gather more data. If more runs confirm the pattern is unchanged (intermittent, not consistent), this revert would reintroduce the original stalling bug without fixing the observed failure.

/close

Due to the cited analysis and improved job performance noted in https://redhat.atlassian.net/browse/TRT-2796?focusedCommentId=17657265

@neisw: Closed this PR.